Towards an Automatic Evaluation for Topic Extraction Systems for Online Reputation Management
نویسندگان
چکیده
This work present a novel evaluation framework for topic extraction over user generated contents. The motivation of this work is the development of systems that monitor the evolution of opinionated topics around a certain entity (a person, company or product) in the Web. Currently, due to the effort that would be required to develop a gold standard, topic extraction systems are evaluated qualitatively over cases of study or by means of intrinsic evaluation metrics that can not be applied across heterogeneous systems. We propose evaluation metrics based on available document metadata (link structure and time stamps) which do not require manual annotation of the test corpus. Our preliminary experiments show that these metrics are sensitive to the number of iterations in LDA-based topic extraction algorithms, which is an indication of the consistency of the metrics. ? This work has been partially supported by Alma Technologies and the Spanish Government (projects Webopinion and Text-Mess/Ines)
منابع مشابه
Automatic keyword extraction using Latent Dirichlet Allocation topic modeling: Similarity with golden standard and users' evaluation
Purpose: This study investigates the automatic keyword extraction from the table of contents of Persian e-books in the field of science using LDA topic modeling, evaluating their similarity with golden standard, and users' viewpoints of the model keywords. Methodology: This is a mixed text-mining research in which LDA topic modeling is used to extract keywords from the table of contents of sci...
متن کاملA collusion mitigation scheme for reputation systems
Reputation management systems are in wide-spread use to regulate collaborations in cooperative systems. Collusion is one of the most destructive malicious behaviors in which colluders seek to affect a reputation management system in an unfair manner. Many reputation systems are vulnerable to collusion, and some model-specific mitigation methods are proposed to combat collusion. Detection of col...
متن کاملRecommendation, trust and reputation management in a group online mentorship system
Existing online mentorship systems typically match mentors and mentees manually. Recommender systems can be used to match mentors and mentees and trust and reputation mechanisms can be used to improve the decision process. This paper discusses the state-of-the-art in online mentorship systems, recommender systems, and trust and reputation mechanisms. It further proposes a five-stage process for...
متن کاملLIA@RepLab 2013
In this paper, we present the participation of the Computer Science Laboratory of Avignon (LIA) to RepLab 2013 edition. RepLab is an evaluation campaign for Online Reputation Management Systems. LIA has produced a important number of experiments for every tasks of the campaign: filtering, topic priority detection, Polarity for Reputation and topic detection. Our approaches rely on a large varie...
متن کاملIdentifying Indicators Affecting the Evaluation of Service Quality of Medical Centers’ Online Appointment Systems
Introduction: Online queuing systems in medical centers significantly reduce waiting time and costs, and increase patient satisfaction with the quality of services provided. Service provisions with the desired quality through these systems will manage the crowds in the health care centers. In the current situation, gatherings cause an upward trend of the COVID-19 pandemic and subsequent problem...
متن کامل